 |
Multimedia API ReferenceSeptember 12, 2016 | 24.2 Release |
|
|
Multimedia API ReferenceSeptember 12, 2016 | 24.2 Release |
This use case application showcases a typical appliance performing intelligent video analytics. Application areas include public safety, smart cities, and autonomous machines. This example demonstrates four (4) concurrent video streams going through a decoding process using the on chip decoders, video scaling using on chip scalar, and GPU compute. For simplicity of demonstration, only one of the channels will use GIE to perform object identification and generate bounding box around the identified object. This sample also uses video converter functions to do various format convrsions and uses EGLImage to demonstrate buffer sharing and image display capability.
In this sample, object detection is limited to identifying cars in video streams of 960 x 540 resolution, running up to 14 FPS. The network is based on GoogleNet. The inference is performed on a frame-by-frame basis and no object tracking is involved. Note that this network is intended to be purely an example to showcase GIE usage to build the compute pipeline quickly. The trained GoogleNet is provided as source and was trained using NVIDIA DIGITS with roughly 3000 frames taken from 5-10 feet elevation. Varying levels of detection accuracy are expected based on the video samples fed in. Given that this sample is locked to perform at half-HD resolutions under 10 FPS, video feeds with higher FPS for inference will show stuttering during playback.
NvEGLImageFromFd is NV-defined API to return an EGLImage pointer from the file descriptor buffer that is allocated by way of Tegra mechanism. EGLImage buffer is then used by GIE to render the bounding box to the image.
This sample does not require a camera.
The image below shows a sample block diagram.
The images below shows data flow details for the channel using GIE.
NvEGLImageFromFd is NVIDIA defined API to return an EGLImage pointer form the file descriptor buffer that is allocated by way of Tegra mechanism. EGLImage buffer is then used by GIE to render the bounding box to the image.
Before running the sample, you must have the following:
There is a global structure struct context_t that manages all the resources in the application.
| Element | Description |
|---|---|
| NvVideoDecoder | Contains all video decoding-related elements and functions. |
| NvVideoConverter | Contains elements and functions for video format conversion. |
| NvEglRenderer | Contains all EGL display rendering-related functions. |
| egl_image | The EGLImage used for CUDA processing. |
| conv_output_plane_buf_queue | Output plane queue for video conversion. |
| dec_capture_loop | The thread handler for decoding capture loop. |
The NvVideoDecoder class creates a new V4L2 Video Decoder. The following table describes the key NvVideoDecoder members that this sample uses.
| Member | Description |
|---|---|
| output_plane | Holds the V4L2 output plane. |
| capture_plane | Holds the V4L2 capture plane. |
| createVideoDecoder | Static function to create video decode object. |
| subscribeEvent | Subscribes event. |
| setExtControls | Sets external control to V4L2 device. |
| setOutputPlaneFormat | Sets output plane format. |
| setCapturePlaneFormat | Sets capture plane format. |
| getControl | TBD |
| dqEvent | Dequeues the devent reported by the V4L2 device. |
| isInError | Checks if under error state. |
The NvVideoConverter class packages all video converting related elements and functions. It performs color space conversion, scaling and conversion between hardware buffer memory and software buffer memory. The following table describes the key NvVideoConverter members that this sample uses.
| Member | Description |
|---|---|
| output_plane | Holds the output plane. |
| capture_plane | Holds the capture plane. |
| waitForIdle | Waits until all the buffers queued on the output plane are converted and dequeued from the capture plane. This is a blocking method. |
| setCapturePlaneFormat | Sets the format on the converter capture plane. |
| setOutputPlaneFormat | Sets the format on the converter output plane. |
Both NvVideoDecoder and NvVideoConverter contains 2 key elements: output_plane and capture_plane. This object is dervied from class type NvV4l2ElementPlane.
NvV4l2ElementPlane creates an NVv4l2Element plane. The following table describes the key NvV4l2ElementPlane members used in this sample.
| Member | Description |
|---|---|
| setupPlane | Sets up the plane of V4l2 element. |
| deinitPlane | Destroys the plane of V4l2 element. |
| setStreamStatus | Starts/Stops the stream. |
| setDQThreadCallback | Sets the callback function of the dqueue buffer thread. |
| startDQThread | Starts the thread of the dqueue buffer. |
| stopDQThread | Stops the thread of the dqueue buffer. |
| qBuffer | Queues a V4l2 buffer from the plane. |
| dqBuffer | Dequeues a V4l2 buffer from the plane. |
| getNumBuffers | Gets the number of the V4l2 buffer. |
| getNumQueuedBuffers | Gets the number of the V4l2 buffer in the queue. |
| getNthBuffer | Gets the NvBuffer queue object at index N. |
GIE_Context provides a series of interfaces to load Caffe model and perform inference. The following table describes the key GIE_Context members used in this sample.
| GIE_Context | Description |
|---|---|
| gie_alloc | Sets up all resources and parameters. |
| gie_free | Clears all resources. |
| caffeToGIEModel | Loads Caffe mode to GIE model. |
| doInference | Interface for inference after GIE model is loaded. |
| parse_bbox | Parses Google net's output results. |
| parse_hel_bbox | Parses Nvhel net's output results. |
| parse_net | Parses net info from proto file. |
2 global functions are used to create and destroy EGLImage from dmabuf file descriptor.
| Global Function | Description |
|---|---|
| NvEGLImageFromFd | Creates EGLImage from dmabuf fd. |
| NvDestroyEGLImage | Destroys the EGLImage. |
To run the sample, execute:
backend <channel-num> <in-file1> <in-file2>... <in-format> [options]
The following video formats are supported for use with command line options:
The table below describes the available options.
| Command | Description |
|---|---|
| -h, –help | Prints the help text. |
| -fps <fps> | Displays rate in frames per second. Default = 30. |
| –s | Provides a statistic of each channel. |
| -run-opt <0-3> | 0=default, 1=parser only, 2=parser + decoder, 3=parser + decoder + VIC |
| –input-nalu | Specifies that the default input to the decoder is in nalu units. |
| –input-chunks | Specifies that the default input to the decoder is in chunk of bytes. |
| –gie-deployfile | Sets the deploy file name. |
| –gie-modelfile | Sets the model file name. |
| –gie-proc-interval | Sets the process interval; 1 frame is processed every gie-proc-interval. |
| –gie-float16 | 0=default, 1=float16, 2=float32. |
For X11 technical details, see:
